Ocular system to optimize learning

ABSTRACT

A method to optimize learning based upon ocular information of a subject includes providing a video camera for recording a close-up view of a subject&#39;s eye. A first electronic display shows a plurality of educational subject matter to the subject. A second electronic display shows an output to an instructor. Changes in ocular signals of the subject are processed through the use optimized algorithms. A cognitive state model determines a low to a high cognitive load experienced by the subject. The cognitive state model is evaluated based on the changes in the ocular signals for determining a probability of the low to the high cognitive load experienced by the subject. The probability of the low to the high cognitive load experienced by the subject is displayed to the instructor.

CROSS-REFERENCE TO RELATED APPLICATIONS

This non-provisional patent application claims priority to provisionalapplication 62/950,918 filed on Dec. 19, 2019, the entire contents ofwhich are fully incorporated herein with these references.

DESCRIPTION Field of the Invention

The present invention generally relates to ocular systems. Moreparticularly, the present invention relates to ocular systems where onecan perform deception detection, assessment of operational risk andoptimized learning, which may be enabled by transillumination of theiris muscles to infer stroma deformation.

Background of the Invention

The inventors of this present application have substantial experience inocular system disclosed by provisional patent application 62/239,840;U.S. Pat. No. 10,575,728 issued on Mar. 3, 2020; and patent applicationSer. No. 16/783,128 filed on Feb. 5, 2020 which is now U.S. Publication2020/0170560—the entire contents of which are fully incorporated hereinwith these references

Accordingly, there is a need for improved ocular systems. The presentinvention fulfills these needs and provides other related advantages.

SUMMARY OF THE INVENTION

Ocular System for Deception Detection

An exemplary embodiment of the present invention is a method ofdeception detection based upon ocular information of a subject, themethod comprising the steps of: providing a standoff device configuredto view the subject during an examination, the standoff device not inphysical contact with the subject, wherein the standoff device has atleast one video camera configured to record a close-up view of at leastone eye of the subject, and wherein the standoff device has or isconnected to a computing device; providing a cognitive state modelconfigured to determine a high to a low cognitive load experienced bythe subject, the cognitive load measuring the extent to which thesubject is drawing on mental resources to formulate their response;providing an emotional state model configured to determine a high to alow state of arousal experienced by the subject, the state of arousalbased upon the subject's nervous system activation; recording, via theat least one video camera, the ocular information of the at least oneeye of the subject; establishing a baseline state of the ocularinformation of the at least one eye of the subject before questioning ofthe subject; asking a question of the subject and allowing the subjectto answer the question; after asking the question and including the timeof the subject answering the question, processing the ocular informationto identify changes in ocular signals of the subject; evaluating, viathe computing device, the cognitive state model and the emotional statemodel based solely on the changes in ocular signals and estimating aprobability of the subject being either truthful or deceptive;determining a binary output of either truthfulness or deceptiveness; anddisplaying the binary output to an administrator.

In other exemplary embodiments the changes in ocular signals maycomprise any of the following: eye movement, gaze location X, gazelocation Y, saccade rate, saccade peak velocity, saccade averagevelocity, saccade amplitude, fixation duration, fixation entropy(spatial), gaze deviation (polar angle), gaze deviation (eccentricity),re-fixation, smooth pursuit, smooth pursuit duration, smooth pursuitaverage velocity, smooth pursuit amplitude, scan path (gaze trajectoryover time), pupil diameter, pupil area, pupil symmetry, velocity (changein pupil diameter), acceleration (change in velocity), jerk (pupilchange acceleration), pupillary fluctuation trace, constriction latency,dilation duration, spectral features, iris muscle features, iris musclegroup identification, iris muscle fiber contractions, iris sphincteridentification, iris dilator identification, iris sphincter symmetry,pupil and iris centration vectors, blink rate, blink duration, blinklatency, blink velocity, partial blinks, blink entropy (deviation fromperiodicity), sclera segmentation, iris segmentation, pupilsegmentation, stroma change detection, eyeball area (squinting),deformations of the stroma, iris muscle changes.

In other exemplary embodiments the step of estimating the probability ofthe subject being either truthful or deceptive comprises a plurality ofestimates taken over a period of time during the subject's answer,wherein the plurality of estimates are weighted and combined to producethe binary output.

In other exemplary embodiments the at least one video camera may captureframes at a rate of at least 100 frames per second, 50 frames per secondor 30 frames per second.

In other exemplary embodiments the standoff device may include a secondvideo camera configured to record the entirety of the subject's face.

In other exemplary embodiments the computing device may be a cloud-basedcomputing device disposed remote from the standoff device.

In other exemplary embodiments the computing device may be part of thestandoff device or may be separate from the standoff device.

In other exemplary embodiments, after asking the question of the subjectand allowing the subject to answer the question, one may wait a periodof time and re-establishing the baseline state of the ocular informationof the at least one eye of the subject before an additional question isasked of the subject.

In other exemplary embodiments an entire statement by the subject may beevaluated as the answer to question.

In other exemplary embodiments the step of saving each binary output andeach corresponding video recorded by the at least one video camera maybe by the computing device.

Ocular System to Assess Operational Risk

An exemplary embodiment of the present invention 1. A method ofassessing operational risk based upon ocular information of a subject,the method comprising the steps of: providing a video camera configuredto record a close-up view of at least one eye of the subject; providingan electronic display screen configured to display a plurality of imagesto the subject; providing a computing device electronically connected tothe video camera and the electronic display; displaying, via theelectronic display, at least one oculomotor task; recording, via thevideo camera, the ocular information of the at least one eye of thesubject during the at least one oculomotor task; processing, via thecomputing device, the ocular information to identify changes in ocularsignals of the subject through the use of convolutional neural networks;evaluating, via the computing device, the changes in ocular signals fromthe convolutional neural networks combined with the at least oneoculomotor task corresponding to the changes in ocular signals by amachine learning algorithm; determining, via the machine learningalgorithm, a duty fitness result for the subject; wherein the dutyfitness result is either fit for duty, unfit for duty or moreinformation needed; and displaying, to the subject and/or to asupervisor, the duty fitness result for the subject.

In other exemplary embodiments the changes in ocular signals maycomprise any of the following: eye movement, gaze location X, gazelocation Y, saccade rate, saccade peak velocity, saccade averagevelocity, saccade amplitude, fixation duration, fixation entropy(spatial), gaze deviation (polar angle), gaze deviation (eccentricity),re-fixation, smooth pursuit, smooth pursuit duration, smooth pursuitaverage velocity, smooth pursuit amplitude, scan path (gaze trajectoryover time), pupil diameter, pupil area, pupil symmetry, velocity (changein pupil diameter), acceleration (change in velocity), jerk (pupilchange acceleration), pupillary fluctuation trace, constriction latency,dilation duration, spectral features, iris muscle features, iris musclegroup identification, iris muscle fiber contractions, iris sphincteridentification, iris dilator identification, iris sphincter symmetry,pupil and iris centration vectors, blink rate, blink duration, blinklatency, blink velocity, partial blinks, blink entropy (deviation fromperiodicity), sclera segmentation, iris segmentation, pupilsegmentation, stroma change detection, eyeball area (squinting),deformations of the stroma, iris muscle changes.

In other exemplary embodiments the at least one oculomotor task maycomprise any of the following: pupillary light reflex, optokineticreflex, horizontal gaze nystagmus, smooth pursuit, gaze calibration orstartle response.

In other exemplary embodiments the electronic display screen may be thatof a smart phone, a tablet, a laptop screen, a desktop screen or anelectronic screen.

In other exemplary embodiments the video camera, the electronic displayscreen and the computing device may all contained as a smart phone or asa tablet.

An exemplary embodiment of the present invention is a method ofassessing operational risk based upon ocular information of a subject,the method comprising the steps of: providing a video camera configuredto passively record a close-up view of at least one eye of the subject;providing a computing device electronically connected to the videocamera and the electronic display; recording, via the video camera, theocular information of the at least one eye of the subject; processing,via the computing device, the ocular information to identify changes inocular signals of the subject through the use of convolutional neuralnetworks; evaluating, via the computing device, the changes in ocularsignals from the convolutional neural networks by a machine learningalgorithm; determining, via the machine learning algorithm, a dutyfitness result for the subject; wherein the duty fitness result iseither fit for duty, unfit for duty or more information needed; anddisplaying, to the subject and/or to a supervisor, the duty fitnessresult for the subject.

In other exemplary embodiments the changes in ocular signals maycomprise any of the following: eye movement, gaze location X, gazelocation Y, saccade rate, saccade peak velocity, saccade averagevelocity, saccade amplitude, fixation duration, fixation entropy(spatial), gaze deviation (polar angle), gaze deviation (eccentricity),re-fixation, smooth pursuit, smooth pursuit duration, smooth pursuitaverage velocity, smooth pursuit amplitude, scan path (gaze trajectoryover time), pupil diameter, pupil area, pupil symmetry, velocity (changein pupil diameter), acceleration (change in velocity), jerk (pupilchange acceleration), pupillary fluctuation trace, constriction latency,dilation duration, spectral features, iris muscle features, iris musclegroup identification, iris muscle fiber contractions, iris sphincteridentification, iris dilator identification, iris sphincter symmetry,pupil and iris centration vectors, blink rate, blink duration, blinklatency, blink velocity, partial blinks, blink entropy (deviation fromperiodicity), sclera segmentation, iris segmentation, pupilsegmentation, stroma change detection, eyeball area (squinting),deformations of the stroma, iris muscle changes.

In other exemplary embodiments the duty fitness result may relate to alevel of intoxication of the subject.

In other exemplary embodiments the duty fitness result may relate to alevel of impairment of the subject.

In other exemplary embodiments the duty fitness result may relate to alevel of fatigue of the subject.

In other exemplary embodiments the duty fitness result may relate to alevel of anxiety and/or stress of the subject.

Ocular System to Optimize Learning

An exemplary embodiment of the present invention is a method to optimizelearning based upon ocular information of a subject, the methodcomprising the steps of: providing a video camera configured to record aclose-up view of at least one eye of the subject; providing a firstelectronic display configured to display a plurality of educationalsubject matter to the subject; providing a second electronic displayconfigured to display an output to an instructor; providing a computingdevice electronically connected to the video camera, the firstelectronic display and the second electronic display; recording, via thevideo camera, the ocular information of the at least one eye of thesubject while learning the plurality of educational subject matter;processing, via the computing device, the ocular information to identifychanges in ocular signals of the subject through the use optimizedalgorithms; providing a cognitive state model configured to determine alow to a high cognitive load experienced by the subject, the cognitiveload measuring the extent to which the subject is drawing on mentalresources; evaluating, via the computing device, the cognitive statemodel based on the changes in the ocular signals and determining aprobability of the low to the high cognitive load experienced by thesubject; and displaying, via the second electronic display, theprobability of the low to the high cognitive load experienced by thesubject to the instructor.

In other exemplary embodiments it may include the steps of, via thecomputing device, establishing a location of the first electronicdisplay in relation to the at least one eye of the subject; determiningfrom the changes in ocular signals a subject's gazing location inrelation to the plurality of educational subject matter; linking thesubject's gaze location of the plurality of the educational subjectmatter and the changes in ocular signals to the subject's cognitiveload; and displaying, via the second electronic display to theinstructor, the subject's cognitive load in relation to the plurality ofeducational subject matter.

In other exemplary embodiments it may include the step of isolating apupil dilation of the subject resulting from changes in cognitive loadfrom changes in ambient luminance by utilizing a power spectral densityfrequency transformation.

In other exemplary embodiments it may include the steps of providing anoptimal learning scale model having a learning scale for the subjectbased upon a representative population or a subject's prior data, thelearning scale ranging from under stimulated to overwhelmed; evaluating,via the computing device, the changes in ocular signals to determine thesubject's position along the learning scale; and displaying, via thesecond display to the instructor, the subject's position along thelearning scale.

In other exemplary embodiments it may include the steps of providing amemory formation model configured to determine a strength of short-termand/or long-term memories; evaluating, via the computing the device, thechanges in ocular signals to determine the subject's strength of theshort-term and/or the long-term memories in relation to the plurality ofeducational subject matter; and displaying, via the second display tothe instructor, the subject's strength of the short-term and/or thelong-term memories in relation to the plurality of educational subjectmatter.

In other exemplary embodiments the changes in ocular signals maycomprise any of the following: eye movement, gaze location X, gazelocation Y, saccade rate, saccade peak velocity, saccade averagevelocity, saccade amplitude, fixation duration, fixation entropy(spatial), gaze deviation (polar angle), gaze deviation (eccentricity),re-fixation, smooth pursuit, smooth pursuit duration, smooth pursuitaverage velocity, smooth pursuit amplitude, scan path (gaze trajectoryover time), pupil diameter, pupil area, pupil symmetry, velocity (changein pupil diameter), acceleration (change in velocity), jerk (pupilchange acceleration), pupillary fluctuation trace, constriction latency,dilation duration, spectral features, iris muscle features, iris musclegroup identification, iris muscle fiber contractions, iris sphincteridentification, iris dilator identification, iris sphincter symmetry,pupil and iris centration vectors, blink rate, blink duration, blinklatency, blink velocity, partial blinks, blink entropy (deviation fromperiodicity), sclera segmentation, iris segmentation, pupilsegmentation, stroma change detection, eyeball area (squinting),deformations of the stroma, iris muscle changes.

In other exemplary embodiments the step of recording, via the videocamera, the ocular information of the at least one eye of the subjectwhile learning the plurality of educational subject matter may alsoinclude recording, via the camera, a facial expression and/or a postureof the subject while learning the plurality of educational subjectmatter.

An exemplary embodiment of the present invention is a method to measurea cognitive load based upon ocular information of a subject, the methodcomprising the steps of: providing a video camera configured to record aclose-up view of at least one eye of the subject; providing a computingdevice electronically connected to the video camera and the electronicdisplay; recording, via the video camera, the ocular information of theat least one eye of the subject; processing, via the computing device,the ocular information to identify changes in ocular signals of thesubject through the use of convolutional neural networks; evaluating,via the computing device, the changes in ocular signals from theconvolutional neural networks by a machine learning algorithm;determining, via the machine learning algorithm, the cognitive load forthe subject; and displaying, to the subject and/or to a supervisor, thecognitive load for the subject.

In other exemplary embodiments the changes in ocular signals maycomprise any of the following: eye movement, gaze location X, gazelocation Y, saccade rate, saccade peak velocity, saccade averagevelocity, saccade amplitude, fixation duration, fixation entropy(spatial), gaze deviation (polar angle), gaze deviation (eccentricity),re-fixation, smooth pursuit, smooth pursuit duration, smooth pursuitaverage velocity, smooth pursuit amplitude, scan path (gaze trajectoryover time), pupil diameter, pupil area, pupil symmetry, velocity (changein pupil diameter), acceleration (change in velocity), jerk (pupilchange acceleration), pupillary fluctuation trace, constriction latency,dilation duration, spectral features, iris muscle features, iris musclegroup identification, iris muscle fiber contractions, iris sphincteridentification, iris dilator identification, iris sphincter symmetry,pupil and iris centration vectors, blink rate, blink duration, blinklatency, blink velocity, partial blinks, blink entropy (deviation fromperiodicity), sclera segmentation, iris segmentation, pupilsegmentation, stroma change detection, eyeball area (squinting),deformations of the stroma, iris muscle changes.

Transillumination of Iris Muscles to Infer Stroma Deformation

An exemplary embodiment of the present invention is a method ofdiscovering relationships between iris physiology and cognitive statesand/or emotional states of a subject, the method comprising the stepsof: providing a computing device; providing a video camera configured torecord a close-up view of at least one eye of the subject; providing afirst light configured to be held to a skin of a lower eyelid of thesubject allowing light to shine out from within the at least one eye;providing a second light configured to not be in contact with thesubject located a distance apart from the subject and configured toilluminate a stroma of the at least one eye of the subject; wherein thefirst light and the second light are electronically synced together andconfigured to flash alternatively; engaging the user in a plurality oftasks, each task of the plurality of tasks configured to be cognitivelyor emotionally evocative; recording, via the video camera, ocularinformation comprising responses in the iris musculature andcorresponding distortions in the stroma due to the cognitive stateand/or the emotional state of the subject produced by the plurality oftasks; processing, via the computing device, the ocular information toidentify correlations between the responses in the iris musculature andthe distortions in the stroma through the use optimized algorithms; andidentifying, via the computing device, at least one predictivedistortion in the stroma capturable solely with a visible-spectrumcamera correlating to a predicted responses in the iris musculature whenthe subject was in the cognitive state and/or the emotional stateproduced by the plurality of tasks.

In other exemplary embodiments the first light may comprise a (150 mw)NIR LED. The second light may comprise a (150 mw) NIR LED.

In other exemplary embodiments the first light and the second light maybe configured to flash alternatively (at 160 Hz) producing a resultanteffect (of 80 Hz).

In another exemplary embodiment, it could further include a method ofgenerating near infrared images from visible light images, the methodcomprising the steps of: providing a visible spectrum video cameraconfigured to record the close-up view of the at least one eye of thesubject; recording, via the visible spectrum video camera, the ocularinformation comprising the distortions in the stroma due to thecognitive state and/or the emotional state of the subject; predicting,via the computing device, an infrared image of the at least one eye ofthe subject through a generative adversarial network using the ocularinformation from the visible spectrum video camera; wherein thepredicting, via the computing device, utilizes the at least onepredictive distortion in the stroma for creating the infrared image.

Other features and advantages of the present invention will becomeapparent from the following more detailed description, when taken inconjunction with the accompanying drawings, which illustrate, by way ofexample, the principles of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings illustrate the invention. In such drawings:

FIG. 1 is a front view of an eye of the subject showing a maskingtechnique based on a UNET neural network;

FIG. 2A illustrates a side view of a camera system of the presentinvention;

FIG. 2B illustrates a schematic top view of a subject utilizing thepresent invention;

FIG. 3 is a flow chart of one embodiment of the present invention;

FIG. 4 illustrates an example of pupillary light reflex test of thepresent invention;

FIG. 5 illustrates an example of optokinetic reflex of the presentinvention;

FIG. 6 illustrates an example of horizontal gaze nystagmus of thepresent invention;

FIG. 7 illustrates an example of smooth pursuit of the presentinvention;

FIG. 8 illustrates an example of gaze calibration of the presentinvention;

FIG. 9 shows one embodiment of the software output of the presentinvention;

FIG. 10 shows another embodiment of the software output of the presentinvention;

FIG. 11 is one embodiment of a cognitive load and learning parameteroutput of the present invention;

FIG. 12 shows one embodiment of the transillumination hardware andprocess;

FIG. 13 is a still image of surface layer stroma and transilluminatediris video as captured with transillumination hardware of FIG. 12 ;

FIG. 14 is an example of GAN generation of IR image for dark eye wherethe generated IR image is used to create the CV mask which is thenprojected onto the visible light image in real time;

FIG. 15 is an example of the CV mask formed using the UNET prediction onthe IR image and overlaid on the visible light image; and

FIG. 16 is a perspective view of one embodiment of a dual camera designcapturing both NIR and visible light of the same location at the sametime.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

It is noted herein that the reference to “Senseye” in the presentapplication is a reference to the company (i.e. the Applicant) of theinventors.

Ocular System for Deception Detection:

The Senseye Deception Detector is a standoff device designed to useocular signals to detect deception in a variety of settings, includingstructured questions, active interrogation, and passively viewing ahuman. The device records ocular signals and classifies a person'sstatement as truthful or deceptive. It provides a binary classification.The classification of each question is based solely on the ocularinformation obtained at the time of the response or statement, andtherefore the system design allows for classification of each questionindividually with no duplicate questions or specific question structurenecessary. This is an advance over many systems and techniques fordetecting deception, which rely on multiple instances of a questiontopic to arrive at a conclusion, or rely on comparing the results ofquestions to each other. The thresholds for deception can be set basedon the use case (e.g., more stringent parameters for higher stakessituations).

The Deception Detector uses a combination of models of cognitive andemotional states to feed into the final deception model andclassification. As such, the system is capable of outputting a binaryclassification of the results of the component models. It outputs aclassification of high or low cognitive load, which measures the extentto which the person is drawing on mental resources to formulate theirresponse. It outputs a classification of high or low arousal, which isbased on the subject's nervous system activation. Both of these measuresare intended to provide context for the classification of deception.

It is also understood by those skilled in the art that the SenseyeDeception Detector could be reconfigured to not be a standoff device andinstead reside, at least partially, in a head gear, hat, pair of glassesand the like that would be worn or held by the user. This manner ofmonitoring and viewing the subject would be more intrusive, but wouldstill use the rest of the methods and strategies as taught herein.

The Deception Detector relies on ocular signals to make itsclassification. These changes in ocular signals may comprise any of thefollowing: eye movement, gaze location X, gaze location Y, saccade rate,saccade peak velocity, saccade average velocity, saccade amplitude,fixation duration, fixation entropy (spatial), gaze deviation (polarangle), gaze deviation (eccentricity), re-fixation, smooth pursuit,smooth pursuit duration, smooth pursuit average velocity, smooth pursuitamplitude, scan path (gaze trajectory over time), pupil diameter, pupilarea, pupil symmetry, velocity (change in pupil diameter), acceleration(change in velocity), jerk (pupil change acceleration), pupillaryfluctuation trace, constriction latency, dilation duration, spectralfeatures, iris muscle features, iris muscle group identification, irismuscle fiber contractions, iris sphincter identification, iris dilatoridentification, iris sphincter symmetry, pupil and iris centrationvectors, blink rate, blink duration, blink latency, blink velocity,partial blinks, blink entropy (deviation from periodicity), sclerasegmentation, iris segmentation, pupil segmentation, stroma changedetection, eyeball area (squinting), deformations of the stroma, irismuscle changes.

The signals are acquired using a multistep process designed to extractnuanced information from the eye. As shown in FIG. 1 , image frames fromvideo data are processed through a series of optimized algorithmsdesigned to isolate and quantify structures of interest. These isolateddata are further processed using a mixture of automatically optimized,hand parameterized, and non-parametric transformations and algorithms.Leveraging the time series character of these signals and the cognitiveload and arousal contextual information, some of these methodsspecifically estimate the probability of the input data representing adeceptive state. Multiple estimates are combined and weighted to producea model that classifies a response or statement, based on the ocularsignals that occur during such, as truthful or deceptive.

The product functions by processing the ocular metrics resulting fromour computer vision segmentation and analyzing these outputs duringtime-linked events occurring in the world. One version of the DeceptionDetector functions on a specific Senseye hardware design. Thisembodiment of the device (see FIG. 2A) has a box with an openingdesigned to sit at eye level standoff from the participant. A videocamera, such as a webcam, is facing the participant captures their head,parsing the eye using facial keypoints. Once the eye location isdetermined, the high resolution camera automatically adjusts to get aclose up view of the left eye (or right eye). As shown in the embodimentof FIG. 2A there is a mirror 2A.1, a USB webcam 2A.2, Canon 70-300 USMAF lens 2A.3, C-Mount to Canon EF Adapter 2A.4, Emergent HR-12000-S-M2A.5 and 10 GigE SFP+ 2A.6.

The camera allows for extremely high fidelity iris segmentation. A highconnection speed allows for over 100 frames per second of update speed,making even the subtlest and quickest changes in eye physiologydetectable. However, slower frame rates can be used such as frame ratesof 50 or 30 frame per second. An adapter mount allows for focal lengthsthat can fill the frame with an eye from over a meter away. In addition,the adapter allows the Senseye system to control the focus ring andaperture via software. Video data is stored in raw format, and processedtabular data is stored in a local database.

One possible use case illustrating the placement and distances of theuser, subject and system is also shown in FIG. 2B. The camera is placedperpendicular to the user and a 45 degree angle mirror is used tocapture the eye. The user can choose to autofocus the lens on aper-session basis. The system alleviates the need for manual focus andremoves a point of human error. As shown in the embodiment of FIG. 2B,the subject 2B.1 is a distance 2B.2 of 36 inches from the fixation point2B.3. The camera 2B.4 is a distance 2B.5 of 24 inches. As shown here theexperimenter 2B.6 is a distance 2B.7 of about 52 inches. Also on thetable is the device 2B.8 of the present invention.

This hardware setup is one of several ways the system is designed towork. The system can also offload its computational workload to anexternal solution such as a cloud instance or an on-site compute node.

In both cases the system functions following the outline in FIG. 3 . Atthe beginning of an assessment the system records metrics which generatea baseline reading of the subject. Many of the metrics listed above andincluded in the model are based on different kinds of change from thesebaseline readings. These changes act as inputs into the model and allowfor an immediate classification of the response as truthful ordeceptive. In between each question, time is allowed for theinterviewee's nervous system and ocular metrics to return to baseline.After this reset period, typically 10 seconds, the interviewer proceedsto the next question and again receives a classification of deception ortruth immediately after the response.

In addition to the immediate classifications, the system outputs anafter session report displaying the results given for each question. Itoffers the option to download the data file containing the readings foreach metric in the model timestamped to the events occuring over theentire session. It has the option to go back and view the video andclassification results of any previously recorded session. The systemhas other features which make it flexible for different use cases. Itprovides the option to create a template of questions, which can beordered and automated for repeated screenings. It can be operated withno template as well, for free questioning. Finally, it can run off ofvideos in which a participant is making statements with no questionsasked. In this case, the entire video statement is viewed as onequestion by the system, and a classification is output in the samefashion, with the same after action options, once the video is complete.

Ocular System to Assess Operational Risk:

The Senseye Operational Risk Management (ORM) System provides anobjective measure of a worker's fitness to work. The system screens theworker for excessive fatigue, alcohol or drug impairment, andpsychological risk factors that could interfere with job performance andsafety. The system records video of the user's eyes while they performvarious oculomotor tasks and/or passively view a screen. The ORM systemalso includes the software that presents the stimuli to the user. Thesystem uses computer vision to segment the eyes and quantify a varietyof ocular features. The ocular metrics then become inputs to a machinelearning algorithm designed to detect when workers are too fatigued orimpaired (due to drugs, alcohol, or psychological risk factors) tosafely perform their job. The thresholds for fitness can be set based onthe use case (e.g., more stringent parameters for high stakes/high riskoccupations). A further application of the ORM models and thresholds isthat they can be implemented on video that passively watches a user asthey are performing a task, with no screening stimuli needed.

The primary input to the Senseye ORM system is video footage of the eyesof the user while they perform the oculomotor tasks presented by thesystem or passively view the phone or tablet screen or computer monitor.The location and identity of visible anatomical features from the openeye (i.e., sclera, iris, and pupil) are classified in digital images ina pixel-wise manner via convolutional neural networks originallydeveloped for medical image segmentation. Based on the output of theconvolutional neural network, numerous ocular features are produced.These ocular metrics are combined with event data from the oculomotortasks which provide context and labels. The ocular metrics and eventdata are provided to the machine learning algorithms which then return aresult of “fit for duty”, “unfit for duty”, or “more informationneeded.” The system will also return the reason behind an “unfit forduty” designation (e.g., excessive fatigue, suspected drug or alcoholimpairment, excessive anxiety).

ORM relies on ocular signals to make its classifications. These changesin ocular signals may comprise any of the following: eye movement, gazelocation X, gaze location Y, saccade rate, saccade peak velocity,saccade average velocity, saccade amplitude, fixation duration, fixationentropy (spatial), gaze deviation (polar angle), gaze deviation(eccentricity), re-fixation, smooth pursuit, smooth pursuit duration,smooth pursuit average velocity, smooth pursuit amplitude, scan path(gaze trajectory over time), pupil diameter, pupil area, pupil symmetry,velocity (change in pupil diameter), acceleration (change in velocity),jerk (pupil change acceleration), pupillary fluctuation trace,constriction latency, dilation duration, spectral features, iris musclefeatures, iris muscle group identification, iris muscle fibercontractions, iris sphincter identification, iris dilatoridentification, iris sphincter symmetry, pupil and iris centrationvectors, blink rate, blink duration, blink latency, blink velocity,partial blinks, blink entropy (deviation from periodicity), sclerasegmentation, iris segmentation, pupil segmentation, stroma changedetection, eyeball area (squinting), deformations of the stroma, irismuscle changes.

The Senseye ORM system is designed to run on a variety of hardwareoptions. The eye video can be acquired by a webcam, cell phone camera,or any other video camera with sufficient resolution and frame rate. Thestimuli can be presented on a cell phone, tablet, or laptop screen or astandard computer monitor. The necessary hardware to run the software isneural-network-capable fpgas, asics or accelerated hardware; eitherwithin the device or on a server accessed through an API.

The Senseye ORM assessment begins with the user initiating the processby logging in to the system. This can be achieved by typing a usernameand password, or using facial recognition. In one embodiment, the useris presented with a series of oculomotor tasks which may include thepupillary light reflex, optokinetic reflex, nystagmus test, and smoothpursuit. A gaze calibration task may also be included to improve thegaze measurements output by the system. Each task is described brieflybelow. Depending on the use case, a subset of these tasks will beincluded. In another embodiment, the scan is designed to be morepassive, so the user's eyes are recorded while they passively view ascreen.

FIG. 4 illustrates an example of pupillary light reflex. The pupillarylight reflex is measured by the ORM system by manipulating the luminanceof the screen. The user fixates a cross in the center of the screenwhile the screen changes from grey to black to white then back to black.The bright white screen causes the pupil to constrict. The pupil size ismeasured using computer vision and various parameters such asconstriction latency, velocity and amplitude are computed. Atypicalpupil dynamics can be indicative of fatigue, intoxication,stress/anxiety, and the sympathetic hyperactivity indicative of PTSD.

FIG. 5 illustrates an example of optokinetic reflex. The optokineticreflex is induced by presenting the user with alternating black andwhite bars moving across the screen. The eye will reflexively track abar moving across the screen and then move back to the starting pointonce the bar has moved off the screen. This produces a characteristicsawtooth pattern in the gaze x position of the eye, which shouldcorrespond to the stimulus velocity. Deviations from the stimulusvelocity indicated that the optokinetic reflex is impaired.

FIG. 6 illustrates an example of horizontal gaze nystagmus. Thenystagmus test is similar to a component of the field sobriety test usedby law enforcement. A circular stimulus moves horizontally across thescreen, traversing 45 degrees of visual angle in each direction. Theuser is instructed to track the stimulus with their eyes. In a healthyindividual, involuntary horizontal eye oscillations are expected tooccur once the eyes have moved 40-45 degrees to the right or left. If auser is intoxicated this involuntary movement will occur at a smallervisual angle.

FIG. 7 illustrates an example of smooth pursuit. This task requires theuser to track a circular stimulus moving at a constant speed with theireyes. Their ability to track the stimulus accurately in terms of speedand spatial precision is quantified. Poor matching of speed or spatiallocation are indicative of impairment.

FIG. 8 illustrates an example of gaze calibration. The gaze calibrationtask consists of a series of dots displayed in 11 different spatiallocations on the screen for a few seconds each. The user is asked tofixate each dot as it appears. This task is used to calibrate the gazetracking system to provide accurate gaze information used to assessbehavior in the other tasks.

Startle response (not illustrated) is when users can be tested withloud, unpredictable bursts of white noise to test their startleresponse. Rapid and large dilations in response to the noise bursts areindicative of sympathetic hyperactivity.

Ongoing with development of ORM models based on the stimuli and metricsdescribed above is their use in passive monitoring situations. In thesecircumstances, the product does not act as a screening device, butrather outputs classification states from the models throughout videoobservation of a user doing a task. These models and thresholds takeadvantage of the same metrics listed above, but are less dependent oncontext due to transfer learning from one scenario to another.

Ocular System to Optimize Learning:

The Senseye Targeted Learning System (TLS) uses non-invasive ocularmeasures of cognitive activity to inform and optimize the process oftraining and skill-based learning. TLS algorithms monitor and classifycognitive events and states, including cognitive effort, short andlong-term memory usage and encoding, and alertness levels. These metricsserve individual purposes as indicators of the cognition required duringa given task. Together, they are able to indicate when a person is in astate conducive to optimal learning. Over time, they are able toquantify a person's learning trajectory. Used in combination with avariety of learning curriculums, TLS aids in adapting curriculumsrapidly to an individual's unique learning pace. This level of adaptivetraining provides accelerated learning while ensuring the retention ofcurriculum material. The targeted learning system includes outputs ofcognitive load, a Senseye Learning Parameter (SLP) and instances ofshort-term and long-term memory encoding.

TLS relies on ocular signals to make its classifications. These changesin ocular signals may comprise any of the following: eye movement, gazelocation X, gaze location Y, saccade rate, saccade peak velocity,saccade average velocity, saccade amplitude, fixation duration, fixationentropy (spatial), gaze deviation (polar angle), gaze deviation(eccentricity), re-fixation, smooth pursuit, smooth pursuit duration,smooth pursuit average velocity, smooth pursuit amplitude, scan path(gaze trajectory over time), pupil diameter, pupil area, pupil symmetry,velocity (change in pupil diameter), acceleration (change in velocity),jerk (pupil change acceleration), pupillary fluctuation trace,constriction latency, dilation duration, spectral features, iris musclefeatures, iris muscle group identification, iris muscle fibercontractions, iris sphincter identification, iris dilatoridentification, iris sphincter symmetry, pupil and iris centrationvectors, blink rate, blink duration, blink latency, blink velocity,partial blinks, blink entropy (deviation from periodicity), sclerasegmentation, iris segmentation, pupil segmentation, stroma changedetection, eyeball area (squinting), deformations of the stroma, irismuscle changes.

The signals are acquired using a multistep process designed to extractnuanced information from the eye. Image frames from video data areprocessed through a series of optimized algorithms designed to isolateand quantify structures of interest. These isolated data are furtherprocessed using a mixture of automatically optimized, handparameterized, and non-parametric transformations and algorithms.

Cognitive Load:

The TLS software is capable of working on any device with a front facingcamera (tablet, phone, computer, VR headset, etc.). The TLS softwareuses anatomical signals (more specifically physiological signals)extracted from images to predict different cognitive states throughoptimized algorithms. The algorithms provide an estimated probabilitythat the input data represents a particular cognitive state, and mayidentify the presence of one or more cognitive states. Image signals arerun through a series of data processing operations to extract signalsand estimations. Multiple image masks are first applied, isolatingcomponents of the eyes as well as facial features allowing variousmetrics to be extracted from the image in real-time. From the imagefilters, pertinent signals are extracted through transformationalgorithms supporting the final estimation of cognitive states. Multipledata streams and estimations can be made in a single calculation, andcognitive load signals may stem from combinations of multiple uniqueprocessing and estimation algorithms. The cognitive load output can bedirectly linked to the stimulus (video and/or images and/or blank screenshown) by relating the events time and time course of the cognitive loadoutput. The software can display, in real-time, the cognitive load ofthe individual as the event is occurring (FIGS. 9 and 10 ).

The TLS product is also capable of utilizing various forms of gaze toperform inference on cognitive states. Gaze used in this product fallsinto three major categories: 1) eye center estimation in frame, 2)estimation of eye position and orientation, and 3) 3D point-of-gazeestimation on the subject's focus point in space. Information cleanedfrom all of these approaches can be used individually or in concert.Individually, these methods offer unique and informative measurements ofeye movement; together (with or without an additional calibrationroutine), they offer cascading informative parameters used to constructa 3D model of the eye and gaze vectors. The point of regard on an objectin real space, such as a computer monitor, can then be estimated byintersecting gaze vectors with a corresponding two-dimensional plane inparallel with the surface of the object. The monitor, IR lights and nIRlights, and camera location are all known quantities before gazeestimation. Gaze of the participant is projected in the form of aheatmap onto the screen the participants are viewing. By plotting thecognitive load at the time of the gaze, the software is able to link thegaze location and the cognitive load associated with the gaze. Thisallows individuals to precisely analyze the location/object/task theparticipant was viewing when there was a change in an individual'scognitive load output.

It is unlikely that the stimulus a user is viewing will exhibit constantluminance. It is well known that perceived changes in ambient luminanceare main drivers of pupillary response. To account for luminance-basedpupillary response, TLS uses power spectral density (PSD) frequencytransformations to isolate pupil dilation resulting from cognitive load.The PSD transformation measures the power of the waveform at eachspecific frequency in an interval. This method can be used to determinethe various types of sinusoids that compose any kind of wave.Deconstruction of the pupillary waveform through PSD has been found todetect cognitive load regardless of luminance condition (Marshall, 2002;Nakayama & Shimizu, 2004; Hampson et al, 2010; Peysakhovich et al.,2015; Peysakhovich et al., 2017; Reiner & Gelfeld, 2014). While theluminance response is reflexive and fast, pupillary changes due tocognitive processes are slower (Joshi et al, 2016). Using a mixture ofmeasured luminance and pupillary response signals, TLS algorithms applyPSD and other transformations, creating new and combinatory signalsderived from multiple time and frequency signals. These signals thendrive probability estimations of cognitive states though optimizedalgorithms, identifying cognitive load states even in the presence ofpupillary responses from external light sources.

Senseye Learning Parameter:

As part of TLS Senseye has developed the Senseye Learning Parameter(SLP). A person's ability to learn can change depending on both internal(e.g. fatigue, low engagement, overwhelmed) and task-related (e.g. tooeasy, too hard) factors. SLP is part of the TLS algorithm that takesinto account individuals' internal factors and is represented as a scalefrom low engagement/understimulated to high internal state/overwhelmed.It is computed using an algorithm which translates an individual'socular signals into a reading on the optimal learning scale, which isstatistically based either on a representative population or anindividual's prior data (see Adaptive Senseye Learning Parameter). Whenthe participant's internal state is low (sustained minimal cognitiveload), the indicator shifts to the low side of the SLP scale while highinternal states (sustained high cognitive load, indicating beingoverwhelmed) will shift the SLP indicator to the high side of the scale.This allows the instructor to adopt and adjust the task so theparticipant can stay in the optimal learning point (middle of the SLP)for best learning results (FIG. 11 ).

Adaptive Senseye Learning Parameter:

As described above, the SLP can operate on a fixed equation to generatean optimal learning parameter. However, it also has the ability tochange its parameters depending on the expertise and learning ability ofthe subject. The amount of stress an individual can undergo while stillabsorbing new information varies from person to person. Under the sameamount of stress and arousal, some people will maintain the ability tolearn while others will not. This variation in cognitive performance atdifferent levels of arousal has been observed in prior research (Chabyet al., 2015; Yerkes and Dodson, 1908; Anderson, 1994). The adaptivefunction of the SLP uses performance to determine the expertise of anindividual (beginning, moderate, expert) and correlates the performancewith the cognitive load to automatically generate an optimal scale forthe individual. The scale is able to shift and adjust depending onchanges in performance and cognitive load of the individual underconditions of stress as the individual learns and masters the task. Thisfunction further enhances the customizability of quantified learning andallows instructors or an automated lesson system to more effectivelymodify the curriculum to individual learning profiles.

Memory Categorization:

The TLS is also able to distinguish the occurrence and strength ofmemory formation including but not limited to the formation ofshort-term memory (STM) and long-term memory (LTM) during the learningprocess. Previous literature shows that different brain regions areinvolved in different types of memory formation. The prefrontal cortexis associated with LTM while the hippocampus is closely associated withSTM. People with lesions or damage in the prefrontal cortex often have adifficult time with memory encoding and retrieval (Jetter et al., 1986;McAndrews and Milner 1991; Eslinger and Grattan 1994; Stuss and others1994; Moscovitch and Winocur 1995) while showing little to no impairmentof short-term memory (Kesner and others 1994; Stuss and others 1994;Swick and Knight 1996; Dimitrov and others 1999; Alexander and others2003). The hippocampus is known to be involved in the formation ofshort-term memory and lesions in the hippocampus impair the encoding ofnew memories (Jonides et al., 2008; Cohen and Eichenbaum, 1993).

The prefrontal cortex is not only involved in LTM but also critical inthe generation of various eye movements through transmission of motorcommands to the brainstem. It is also known to modulate pupil diameterchanges (Schlag-Ray et al., 1992; Ebitz and Moore, 2017) which have beenassociated with memory formation and retrieval (Kucewicz et al., 2018).Because both LTM and ocular metrics are associated with the prefrontalcortex, we can utilize ocular metrics to read out memory formation andbuild a model based on the different patterns of ocular metrics thatoccur during the formation of LTM and STM. Using this model, TLS hasbuilt a feature that outputs the strength and type of memory formationthat occurs while a person is engaged in a learning task.

FIGS. 9 and 10 show one embodiment of TLS software output of thecognitive load session. These figures represent an iteration of the TLSuser interface. FIG. 9 is a live time-series analysis of the user'scognitive load and SLP output during a specific session. In FIG. 10there are instantaneous feeds for both cognitive load and SLP and livefeeds of the eye detection and masking algorithms used to output therespective features. The eye detection algorithm locates both eyes on auser's face and focuses on one eye to segment the iris, pupil andsclera. The instantaneous readout of cognitive load is reported visuallyin the form of a gauge and quantitatively in the form of a percent orsimilar interpretation of the user's cognitive load.

FIG. 11 is one embodiment of the UI/UX of TLS cognitive load and senseyelearning parameter output. One representation of cognitive load leveland learning state along the SLP. The grey dot is the current state ofthe participant. On top is the cognitive load meter where low cognitiveload is represented by green area while the high cognitive load area ispresented in red. The bottom is the SLP. Similar to the cognitive loadreadout, the grey dot represents the current learning state of theparticipant. The extreme ends of the scale represent sub-optimallearning states, due to under and over stimulation.

Transillumination of Iris Muscles to Infer Stroma Deformation:

As a general overview, previous literature has shown that the irismuscles of the eye are innervated by specific regions of the brain.Activation of these brain areas results in complementary changes withinrespective muscle groups of the iris, and has led to the hypothesis thatiris physiology can provide a non-invasive means to quantify relevantcognitive states. Notably, direct observation of iris muscle physiologyis obscured by an overlying membrane known as the stroma. The techniqueoutlined here, henceforth referred to as “transillumination”, is amethod for visualizing iris muscle physiology and anatomy, andsubsequently visualizing how these patterns of muscle movement manifestas distortions within the overlying stroma of the iris. By mapping theassociation of known muscle physiology with known patterns of stromadistortion, transillumination enables the user to infer complex patternsof iris muscle physiology from simple surface level video recordings ofthe eye. Transillumination is an integral technology for accessing brainsignals from the eye.

Senseye has developed a technique for analyzing the contributions ofindividual muscle groups and fibers to movements of the iris associatedwith cognitive and emotional states, and for mapping these movementsonto the surface layer of the eye, the stroma, which is visible tooff-the-shelf cameras. This innovation is a novel and sizeable steptowards achieving a contact-free method of reading brain activity fromocular metrics. It involves both a conceptual innovation, and atechnical innovation. The conceptual innovation is in looking at theindividual movements of the muscles under different cognitive states toextract reliable signal. The technical innovation is a method by whichthe stroma and transilluminated muscles are visualized in such a way asto be able to be mapped onto each other.

The muscles of the iris are innervated by the parasympathetic andsympathetic nervous system. Specifically, the dilator muscles of theiris are innervated by many individual termini of the SNS, and thesphincter muscles are innervated by many individual termini of the PNS.These innervations allow information along those nervous systems'pathways to travel downstream to individual muscles of the iris, causingmovements that can be measured and used to infer cognitive and emotionalstates. The transillumination technique of viewing and mapping the irismuscles onto the stroma allows for the creation of Senseye products thatuse surface level changes in the iris to model brain activity.

In regards to the process, the signal acquisition device consists of twolighting components, henceforth referred to as “lighting component one”(LC1) and “lighting component two” (LC2), and one camera. LC1 is asingle 150 mw nIR LED powered by 5 volts. This is held to the skin ofthe lower eyelid in a manner that allows the light to shine out fromwithin and render the musculature of the iris visible. LC2 is a 150 mwnIR standoff LED array that illuminates the exterior stroma of the iris(FIG. 1 ). These lighting systems are synced to flash alternatingly at160 hz each using an oscilloscope, producing the effect of two 80 hzvideos, one of the iris musculature, and one of the exterior stroma,which are perfectly in sync (FIG. 2 ). Video data is collected using acamera with a frame rate and resolution capable of capturing the finemovements of the iris muscles.

The data collection protocol places the participant in a seat in frontof the camera while an automated series of directions and tasks ispresented (FIG. 12 ). The application has configurable timings for eachtask, changes images on the screen in front of the participant in orderto induce ocular response, and provides automated instruction as to whento perform certain behaviors that are cognitively or emotionallyevocative, such as arithmetic or placing one's hand in ice. Theapplication outputs specific time stamped events in the form of tabulardata to allow recorded footage to be synced with produced stimuli forproper analysis.

In the next series of analyses, image frames from video data areprocessed through a series of optimized algorithms and transformationsdesigned to isolate and quantify structures of interest. Data derivedfrom images illuminated by LC1 is used to parse structures from directobservations of iris musculature. Data derived from images illuminatedby LC2 is used to parse distortions within the overlying stroma of theiris. The resulting image pairs provide unique surface-to-subsurfacemapping of involuntary iris muscle actions. Extracted signals from theseimages are collected in a structured format and stored with pertinentexperimental metadata capable of contextualizing a wide range ofcognitive states and processes. These novel data sets can be used to mapbrain activity and surface stroma movements directly to subsurface irisactivity in a measurable, reliable way.

FIG. 12 shows one embodiment of the transillumination hardware andprocess. One can see the participant holding an LED to eye while alsoilluminated by LEDs on monitor, alternating at 160 hz. The camera facingthe participant capturing video of an eye.

FIG. 13 is a still image of surface layer stroma and transilluminatediris video as captured with transillumination hardware. 13.1 is pointingto the transilluminated sphincter muscles, 13.2 is pointing to thetransilluminated dilator muscles and 13.3 is pointing to the surfacelayer stroma.

Method for Generating NIR Images from RGB Cameras

The method of the present invention is using generative adversarialnetworks and a combination of visible and IR light which are now furtherdiscussed. Continuing the theme of creating a mapping between subsurfaceiris structures visible in IR light onto surface structures seen invisible light, Senseye has developed a method of projecting iris masksformed on IR images onto the data extracted from visible light. Thistechnique uses a generative adversarial network (GAN) to predict the IRimage of an input image captured under visible light (see FIG. 14 ). TheCV mask is then run on the predicted IR image and overlaid back to thevisible light image (see FIG. 15 ).

Part of this method is generating a training set of images on which theGAN learns to predict IR images from visible light images (see FIG. 14). Senseye has developed a hardware system and experimental protocol forgenerating these images. The apparatus consists of two cameras, onecolor sensitive, and one NIR sensitive (see numerals 16.1 and 16.2 inFIG. 16 ). The two are placed tangent to one another such that a hotmirror forms a 45 degree angle with both (see numeral 16.3 in FIG. 16 ).The centroid of the first surface of the mirror is equidistant from bothsensors. Visible light passes straight through the hot mirror onto thevisible sensor and NIR bounces off into the NIR sensor. As such, thesystem creates a highly optically aligned NIR and color image which canbe superimposed pixel-for-pixel. Hardware triggers are used to ensurethat the cameras are exposed simultaneously with error<1 uS.

FIG. 16 is a diagram of hardware design that captures NIR and visiblelight video simultaneously. Two cameras, one with a near IR sensor andone with a visible light sensors are mounted on a 45 degree anglechassis with a hot mirror (invisible to one camera sensor, and an opaquemirror to the other) to create image overlays with pixel-level accuracy.

Creating optically and temporally aligned visible and NIR datasets withlow error allows Senseye to create enormous and varied datasets that donot need to be labelled. Instead of manual labelling, the alignmentallows Senseye to use the NIR images as reference to train the colorimages against. Pre-existing networks already have the ability toclassify and segment the eye into sclera, iris, pupil, and more, givingus the ability to use their outputs as training labels. Additionally,unsupervised techniques like pix-to-pix GANs utilize this framework tomodel similarities and differences between the image types. These dataare used to create surface-to-surface, and/or surface-to-subsurfacemapping of visible and invisible iris features.

Other methods being considered to properly filter the RGB spectrum so itresembles the NIR images, is the use of a simulation of the eye so thatrendered images resembles both natural light and that in NIR lightspectrum. The neural network structures would be similar to those listedpreviously (pix-to-pix) and the objective would be to allow for the subcornea structures (iris and pupil) to be recovered and segmentedproperly despite the reflections or other artifacts caused by theinteraction of the natural light spectrum (360 to 730 nm) with theparticular eye.

The utility of the GAN is to learn a function that is able to generateNIR images from RGB images. The issues with RGB images derive from thedegradation of contrast between pupil and iris specifically for darkereyes. What this means is that if there isn't enough light flooding theeye, the border of a brown iris and the pupil hole are indistinguishabledue to their proximity in the color spectrum. In RGB space, because wedo not control for a particular spectrum of light, we are at the mercyof another property of the eye which is that it acts as a mirror. Thisproperty allows for any object to appear as a transparent film on top ofthe pupil/iris. An example of this is you can make out a smaller versionof a bright monitor on your eye given an rgb image. So the GAN acts as afilter. It filters out the reflections, sharpens boundaries, and due toits learned embedding, it is capable of restoring the true boundary ofiris and pupil.

Although several embodiments have been described in detail for purposesof illustration, various modifications may be made to each withoutdeparting from the scope and spirit of the invention. Accordingly, theinvention is not to be limited, except as by the appended claims.

What is claimed is:
 1. A method to optimize learning based upon ocularinformation of a subject, the method comprising the steps of: providinga video camera configured to record a close-up view of at least one eyeof the subject; providing a first electronic display configured todisplay a plurality of educational subject matter to the subject;providing a second electronic display configured to display an output toan instructor; providing a computing device electronically connected tothe video camera, the first electronic display and the second electronicdisplay; recording, via the video camera, the ocular information of theat least one eye of the subject while learning the plurality ofeducational subject matter; processing, via the computing device, theocular information to identify changes in ocular signals of the subjectthrough the use optimized algorithms; providing a cognitive state modelconfigured to determine a low to a high cognitive load experienced bythe subject, the cognitive load measuring the extent to which thesubject is drawing on mental resources; evaluating, via the computingdevice, the cognitive state model based on the changes in the ocularsignals and determining a probability of the low to the high cognitiveload experienced by the subject; and displaying, via the secondelectronic display, the probability of the low to the high cognitiveload experienced by the subject to the instructor.
 2. The method tooptimize learning of claim 1, including the steps of, via the computingdevice, establishing a location of the first electronic display inrelation to the at least one eye of the subject; determining from thechanges in ocular signals a subject's gazing location in relation to theplurality of educational subject matter; linking the subject's gazelocation of the plurality of the educational subject matter and thechanges in ocular signals to the subject's cognitive load; anddisplaying, via the second electronic display to the instructor, thesubject's cognitive load in relation to the plurality of educationalsubject matter.
 3. The method to optimize learning of claim 1, includingthe step of isolating a pupil dilation of the subject resulting fromchanges in cognitive load from changes in ambient luminance by utilizinga power spectral density frequency transformation.
 4. The method tooptimize learning of claim 1, including the steps of providing anoptimal learning scale model having a learning scale for the subjectbased upon a representative population or a subject's prior data, thelearning scale ranging from under stimulated to overwhelmed; evaluating,via the computing device, the changes in ocular signals to determine thesubject's position along the learning scale; and displaying, via thesecond display to the instructor, the subject's position along thelearning scale.
 5. The method to optimize learning of claim 1, includingthe steps of providing a memory formation model configured to determinea strength of short-term and/or long-term memories; evaluating, via thecomputing the device, the changes in ocular signals to determine thesubject's strength of the short-term and/or the long-term memories inrelation to the plurality of educational subject matter; and displaying,via the second display to the instructor, the subject's strength of theshort-term and/or the long-term memories in relation to the plurality ofeducational subject matter.
 6. The method to optimize learning of claim1, wherein the changes in ocular signals comprise any of the following:eye movement, gaze location X, gaze location Y, saccade rate, saccadepeak velocity, saccade average velocity, saccade amplitude, fixationduration, fixation entropy (spatial), gaze deviation (polar angle), gazedeviation (eccentricity), re-fixation, smooth pursuit, smooth pursuitduration, smooth pursuit average velocity, smooth pursuit amplitude,scan path (gaze trajectory over time), pupil diameter, pupil area, pupilsymmetry, velocity (change in pupil diameter), acceleration (change invelocity), jerk (pupil change acceleration), pupillary fluctuationtrace, constriction latency, dilation duration, spectral features, irismuscle features, iris muscle group identification, iris muscle fibercontractions, iris sphincter identification, iris dilatoridentification, iris sphincter symmetry, pupil and iris centrationvectors, blink rate, blink duration, blink latency, blink velocity,partial blinks, blink entropy (deviation from periodicity), sclerasegmentation, iris segmentation, pupil segmentation, stroma changedetection, eyeball area (squinting), deformations of the stroma, irismuscle changes.
 7. The method to optimize learning of claim 1, whereinthe step of recording, via the video camera, the ocular information ofthe at least one eye of the subject while learning the plurality ofeducational subject matter also includes recording, via the camera, afacial expression and/or a posture of the subject while learning theplurality of educational subject matter.